Global Climate Change¶
A Data Science Tutorial by Jamie Lebovics¶
According to NASA$_1$, at least 97% of scientists believe that global climate change is real, and that it is caused by human activity. Yet, it is still considered a controversial debate topic, and many people are misinformed, and people in power are not adequately changing policies to alleviate the problem that our planet is facing, and by extension, the entire population is facing.
This tutorial will show how to look at some existing data about the global and United States climate and visualize it in a way that meaningfully conveys that climate change is real. It is important to continue to do analyses like this one in order to continue to educate those who may be misinformed because of the other 3% of scientists who have gotten a disproportionate voice, making people doubt the need for policy changes aimed at stopping climate change.
Further, I'll explore at how different parts of the world may be affected differently.
1 - source and further reading: https://climate.nasa.gov/scientific-consensus/
Data Collection¶
I downloaded climate data from Kaggle$_2$, which is behind a login and therefore could not be included directly in this tutorial, and wrote the data to a pandas DataFrame. Then, since I also need information about where each country is bycontinent and region, I accessed the a raw table with this information that has been hosted publicly on github$_3$
2 - https://www.kaggle.com/berkeleyearth/climate-change-earth-surface-temperature-data
3 - https://githubusercontent.com/lukes/ISO-3166-Countries-with-Regional-Codes/master/all/all.csv
import requests
url = "https://raw.githubusercontent.com/lukes/ISO-3166-Countries-with-Regional-Codes/master/all/all.csv"
r = requests.get(url)
# display just the first 1000 characters to get an idea of what the page looks like
r.text[:1000]
Data Processing¶
In this section, I'll manipulate the data from the two sources to be usable later in. That means adding columns for the year and the month to DataFrame made from the kaggle data, integrating the region information into that main DataFrame, and calculating the averages of the data over each year and integrating that data into the DataFrame. There will be more subtle data manipulating later on as well, but the main work happens here.
First, I'll import package necessary for managing the data. Pandas holds the data in an easy-to-manipulate DataFrame.
import pandas
The table with region info on each country has to be converted from a csv to a pandas DataFrame. There are commas in some of the country names which have to be specially accounted for since the columns are also separated by columns.
# this should have been a one-line list comprehension but some countries have a comma in their names and quotes
# instead of an escape character so i needed to account for the special case
regions = []
for row in r.text.split('\n'):
if '\"' not in row:
regions.append([col for col in row.split(',')])
else:
lst = [col for col in row.split(',')]
name = (lst[0]+", "+lst[1]).strip('\"')
lst = lst[2:]
lst.insert(0,name)
regions.append(lst)
regions = pandas.DataFrame(regions[1:], columns=regions[0])
regions.set_index('name', inplace=True)
regions.head()
Next, process the climate data by creating a pandas DataFrame, and adding columns with year information, derived from the string date column 'dt'
data = pandas.DataFrame.from_csv('GlobalLandTemperaturesByCountry.csv', index_col=None)
year = [int(dt[:4]) for dt in data['dt']]
month = [int(dt[5:7]) for dt in data['dt']]
data['year'] = year
data['month'] = month
# Impute rows without temperatures listed
data = data[data['AverageTemperature'].isnull()==False]
data.head()
Calculate the yearly averages for each country and add them to the DataFrame
intermediate = data.groupby(['Country', 'year']).mean().reset_index()
intermediate.rename(columns={'AverageTemperature': 'YearlyAverageTemperature', \
'AverageTemperatureUncertainty': 'YearlyAverageTemperatureUncertainty'}, inplace=True)
intermediate = intermediate.drop('month', axis=1)
data = pandas.merge(data, intermediate, how='right', on=['Country', 'year'])
data.head()
Add the region data (only the region and sub-region columns) into the main DataFrame.
intermediate = []
for country in data['Country'].unique():
for c in regions.itertuples():
if country in c.Index:
intermediate.append([country, c.region, c._6])
df = pandas.DataFrame(intermediate, columns=['Country','region','sub_region'])
data = pandas.merge(data, df, how='right', on=['Country'])
data.head()
Exploratory Analysis & Data Visualization¶
Next, I'll plot some of the data what has been collected, processed, and compiled into a single DataFrame. There will be plots for the United States, the world divided by region, and each continent divided by sub-region.
import matplotlib.pyplot as plt
from ggplot import *
import numpy as np
import warnings
warnings.filterwarnings('ignore') # for fewer visual distractions
Below, the plot for the the United States clearly shows a gradual rise in temperature.
us_data = data[['Country','year','YearlyAverageTemperature','YearlyAverageTemperatureUncertainty']].drop_duplicates()\
[data['Country']=="United States"]
ggplot(aes(x='year', y='YearlyAverageTemperature'), data=us_data) +\
geom_point() +\
ggtitle("United States Average Temerature over time") +\
xlab('Year') +\
ylab('Average Temperature (Celsius)') +\
stat_smooth(span=0.2)
Next, we'll look at global data, and see if the world regions are useful ways of grouping climates.
global_data = data[['Country','year','YearlyAverageTemperature','YearlyAverageTemperatureUncertainty',\
'region','sub_region']].drop_duplicates()
for r in data['region'].unique():
g = ggplot(aes(x='year', y='YearlyAverageTemperature', color='sub_region'), \
data=global_data[global_data['region']==r]) +\
geom_point() +\
ggtitle("%s Average Temerature over time"%(r)) +\
xlab('Year') +\
ylab('Average Temperature (Celsius)') +\
stat_smooth(colour='blue', span=0.5)
print(g)
It turns out the temperatures don't cluster well by region. Since climate and global temparature patterns are quite complex, it would be interesting to explore further and see what would be a good way to group temperature by locations, and I'll be doing exactly that in the next section.
Machine Learning¶
The goal in this section is to find a new set of regions that better predicts the tempaerature trends of those countries within that region. Some large countries, like the United States, Russia, and Australia would probably benefit from being divided further into smaller regions to anylize, but for the simplicity I'll consider a country to be the smallest geographic unit.
Some of the code was adapted from the example available through scikit-learn$_{4,5}$. The sci-kit learn documentation is a great resource for learning about K-Means and other machine learning approaches. There is a helpful tutorial that goes in depth on K-Means which I have linked to below$_6$
First, import the additional packages which will be necessary. I'll be using scikit-learn, as previously mentioned, anlong with scipy for some additional work. I'll also be using numpy here, which was imported earlier.
4 - http://scikit-learn.org/stable/auto_examples/cluster/plot_cluster_iris.html
from sklearn.cluster import KMeans
import scipy
Next, prepare the data as a numpy array. Use scikit-learn's K-Means tools to find the clusters for a cluster size of 15. I chose this number after trying out different numbers and finding that this one gave relatively good results. The plot shows the clusters grouped by color.
X = np.array(data[['AverageTemperature','month']])
# normalize the data
X = scipy.stats.zscore(X)
# multiply Temperature col by 100 so it gets weighted more heavily
X[:,0] *= 100
y = np.array(len(data)*[0])
est = KMeans(n_clusters=15)
fig = plt.figure(1, figsize=(4, 3))
plt.clf()
fig, ax = plt.subplots()
plt.cla()
est.fit(X)
labels = est.labels_
ax.scatter(X[:, 0], X[:, 1], c=labels.astype(np.float))
ax.set_xlabel('Temp Z-score*100')
ax.set_ylabel('Month Z-score')
plt.show()
Now that there are new region labels, add those to the main DataFrame so that the data can be plotted by region. For each region, show it overall with a smoothed conditional mean in red on top, and then show that same data sorted by country. Notice the countries in each region.
data['new_region'] = est.labels_
global_data = data[['Country','year','YearlyAverageTemperature','YearlyAverageTemperatureUncertainty',\
'new_region']].drop_duplicates()
for r in data['new_region'].unique():
g = ggplot(aes(x='year', y='YearlyAverageTemperature'), \
data=global_data[global_data['new_region']==r]) +\
geom_point() +\
ggtitle("Region #%s Average Temerature over time"%(r)) +\
xlab('Year') +\
ylab('Average Temperature (Celsius)') +\
stat_smooth(color='red', span=0.5)
print(g)
g = ggplot(aes(x='year', y='YearlyAverageTemperature',color='Country'), \
data=global_data[global_data['new_region']==r]) +\
geom_point() +\
ggtitle("Region #%s Average Temerature over time"%(r)) +\
xlab('Year') +\
ylab('Average Temperature (Celsius)')
print(g)
Insight & Policy Decision¶
The graphs produced by K-means are still not perfect regional breakdowns, but they do provide a different picture, where regions can be determined by climate and season, rather than arbitrary geographic lines. Within each region, you can see a different slope as the average temperature changes over time, because different regions are affected differently. A couple have even gotten cooler over time. It's called "Global Climate Change," and not "Global Warming," because as is evident, not every region is warming up, and certainly not all at the same rates. The Global climate is shifting and changing, affecting different areas differently. For more information, consider reading about the Eath's Climate System$_7$ and even more evidence of climate change by NASA$_8$.
With this knowledge about global climate change, we can advocate for further research into how different regions are affected, and what efforts in each region would best mediate the effects that have already been set in motion.